Model - Based Intrusion Detection System Design and Evaluation

Eighteen years after the original Internet worm of 1988, software still suffers from vulnerabilities that allow attackers to gain illicit access to computer systems. Attackers exploit vulnerabilities to hijack control of a process’ execution as a means to access or alter a system as they desire. In this dissertation, we argue that model-based anomaly detectors can retrofit efficient attack detection ability to vulnerable programs. These detectors restrict a process’ execution using a precomputed model of normal, expected behavior. We construct models of behavior using static binary analysis. While previous statically-constructed models have traded attack detection ability for performance, our new Dyck model is the first statically-constructed model that balances security and performance, and it demonstrates that the previous trade-off was not a fundamental limitation of static analysis. We further improve the Dyck model by incorporating into the model information about data values used in the program and about the execution environment in which the program runs. We quantify such improvements with a new evaluation metric for complex program models. We then attack program models. We automatically discover mimicry and evasion attacks that avoid detection by hiding malicious activity within valid behavior allowed by the model. We start with two models: a program model of the application’s execution behavior and a model of securitycritical operating system state. Given unsafe OS state configurations that describe the goals of an attack, we then find behaviors allowed as valid execution by the program model that produce the unsafe configurations. Our goal is to show that a program model allows no malicious behavior, or to find particular weaknesses in the model. Our work demonstrates the viability of model-based anomaly detection. Although the vulnerabilities of the past eighteen years may persist, model-based anomaly detection provides a mechanism to prevent attackers exploiting a vulnerability from accessing or damaging the system.